Point of recordation terminal apparatus and method of operation

ABSTRACT

A system for providing video intelligence as a service to a plurality of small and medium sized enterprises to distributed video recordation equipment, comprising an apparatus and computer-implemented method. An apparatus comprises a point of recordation terminal apparatus coupled to a public network. A method for operating a system for video intelligence in a public network in a secure manner.

This application is a division of pending application Ser. No.12/395,437 filed Feb. 27, 2009 and claims the benefit of the prior-filedapplication under 35 U.S.C. 120, 121, or 365(c)e.

BACKGROUND

Security cameras are increasingly important for both enterprises andconsumers. All levels of government are promoting installation ofcameras to address fears of crime. Liability insurers may raise rates oncustomers who cannot document that their premises are controlled. Butthe market is bifurcated into extremely costly high end integratedservices and low cost do-it-yourself system design projects forhobbyists. By high complexity image sequences the present inventionincludes high resolution digital photographs, lower resolution movingimages in the form of a series of video frames, meta-data about thetime, place, and conditions of the image, and derived data fromquantitative metrics of the images and compressed low resolutionextracts from images. Internet Protocol (IP) network digital cameras areknown as an accepted solution for security and monitoring. Utilizing IPnetworks instead of dedicated video connections to a local serverdramatically improves system flexibility and can reduce connectivity andmanagement complexity.

Conventional IP network camera system design requires “logging in” toeach camera. Typically, each camera implements a website for useraccess. After a user connects to a camera, he or she may then view data,configure the camera, control conventional camera pan, tilt, and zoom(PTZ) functions, or view a real time stream of image data. In commonapplications, people also want to record the video to allow an analysisof events either missed in real time or not observed with the necessaryattention.

Conventional cameras can be configured to send an email including imageswhen an event happens. Conventional cameras can be configured tobroadcast or stream video. Conventional cameras can be configured toperform a file transfer protocol (FTP) transaction, in a non-limitingexample, uploading at least one image. While this is closer to a desiredend user functionality, conventional implementations require extensivenetwork application and system engineering and only result in transferof limited amounts of information. For example it is observed by theinventors that configuration of each network environment consists atleast of opening ports, mapping addresses, managing a difficultmaintenance and operations model to be assured that the system isworking when needed, and addressing security concerns. For example, isthe equipment on premises vulnerable to theft or damage, can end usersproperly configure the network and the specific camera device, whatsteps are needed to easily record and analyze the video.

To allow live access to cameras, a user should be able to configurefirewalls if external access is to be allowed and to configure an IPaddress resolution service such as a dynamic DNS application. Becausethe solution depends on an occasional user to define and configure eachsecurity installation, deployed solutions have been known to exhibitvery poor security such as unintended publicly viewable webcams.

It is known that configuring for recording video is even more complexthan simply viewing it. The typical solution requires selecting andinstalling an additional system into the user's local network to recordthe video, configuring the cameras to transmit incoming data in a mannercompatible with the recording system, and assuring all networkconfigurations are correct to allow reliable communication betweencameras and recording systems. This introduces additional hardware to beconfigured and maintained. It creates an additional exposure for assetsto be stolen or damaged. Prior to beginning the installation, users mustdetermine how large and complex a system they will ultimately require orsome procurement will turn out to be inadequate and soon obsolete.

To utilize outbound FTP functionality, the user of conventional systemsmust configure a server to accept the FTP transactions and configure thecamera to upload the data appropriately. Further, since the FTPtransaction is typically not in real time, the size is limited by theamount of memory available for storage on the device. Alternately anemail solution can be considered. Unfortunately, e-mail cannot typicallyprovide true video recording. Limitations of email servers and emailaccounts constrain the email alert model to only a few images. Further,since email does not enable realtime streaming of data to the emailserver, the total size of the stored video is limited to the storage onthe device.

Conventional video security systems do not enable proactive monitoringof their status. End users occasionally discover when an event occurs intheir premises, that their system was not functioning correctly and thatthey do not have the desired critical information despite having madeinvestments into both cameras and recording systems. Since videomonitoring systems are typically not core to the business of mostenterprises, but supportive, the resources allocated to maintain thesystem are frequently inadequate, insufficient, or lack the properexpertise to maintain the system effectively. This results in many videosystems being effectively turned off after a period of time as the costand complexity of maintaining the system overwhelms the day to daybenefits. Only the largest governmental or private enterprises havecontinuous human monitoring of all cameras.

The challenge of maintaining operational systems has been addressed inother domains effectively by adopting a “service model” where minimalequipment is onsite and a centralized service provides functionality toa large pool of users. Video monitoring has historically been unable touse this model effectively due to the high bandwidth required toeffectively record usable quality video. While this bandwidth can beaddressed in local area networks, a service model with centralizedrecording requires video to be sent over a wide area network such as theInternet, and such connection may be costly and typically limited. Forexample many business have traditionally had “T1” connectivity, which isbidirectional at about 1 megabit per second. A single camera with highquality video in traditional implementations uses 2-3 megabits ofbandwidth, making a conventional service based model impractical.

The benefits of a service based model would be significant. One keybenefit is the ability to use shared resources across a larger number ofcustomers. This amortizes the cost of equipment, monitoring andmaintenance, allowing very high levels of service at manageable costs.In the area of equipment and management, it is known a single logicalstorage volume, potentially made up of a very large number of physicalvolumes, can be shared amongst a large number of users if there aresufficient safeguards for privacy. Using a single large logical storagevolume allows for significant individual variance in usage patterns tobe efficiently addressed. A single large logical storage volume alsoallows additional reliability and maintenance investments to beamortized over the entire user set, significantly increasing reliabilityand reducing costs.

Similarly it is known that a set of processing elements can beefficiently shared amongst a plurality of sporadic processing demands.The virtual machine model is one well known implementation that allowsprocessing to be allocated and de-allocated to processing resources ondemand. Several other processing models are known ways of distributingcomputational demands over a large number of processing elements. Themodels include pipelining, where a single processing element performs asmall part of the overall function for multiple processing demands, andthreading, where a single process is divided into multiple logicalsubprocesses.

These processing and storage models have been optimized in acomputational architecture commonly called “cloud computing”. In cloudcomputing a very large number of machines and a very large amount oflogical storage is made available in an on-demand basis to a large bodyof customers. Customers can increase and decrease the amount ofcomputational resources allocated to them on a demand basis. Eachcomputation resource is some version of a virtual machine, which canthen be further partitioned into individual user computation needs asoutlined above. Cloud computing also provides cloud storage, where avery large amount of storage is made available on a demand basis,allowing customers to allocate and de-allocate storage as needed. Oneexample of cloud computing is Amazon's Elastic Computing Cloud (EC2).One example of cloud storage is Amazon's Simple Storage Service (S3).

The following processes are known in the art as methods for motiondetection: processing a constant sequence of images (video),establishing a reference image of the scene with only background items,detecting when pixels are changed sufficiently in subsequent images toindicate areas in motion, counting the number of pixels in motion todetermine if enough have changed to indicate an event of interest, andupdating the background image for areas that have changed minimally.Significant improvements are known on this basic algorithm includingobject detection and object recognition. Thus it can be appreciated thatwhat is needed is an apparatus which makes deployment, maintenance, andoperation of IP network cameras much less complex. What is needed isequipment that is extremely easy to set up and maintain by using a cloudcomputing infrastructure and strategy.

SUMMARY OF THE INVENTION

A novel implementation of a security camera, is a Point of RecordationTerminal (PORT) apparatus disclosed as follows. In use, a plurality ofpoint of recordation terminals (PORTs) are distributed among small andmedium sized enterprises for installation in their respective privatenetworks. Each PORT captures and analyzes images to determine if thereis an event of interest. Events of interest are compressed, formattedand stored to construct an asset. A reference to each asset istransmitted in near real-time comprising a compressed single frame,time, date, meta-data associated with the assets not transmitted andidentity of the terminal. The reference provides sufficient informationto uniquely access the associated asset on the specific PORT. The PORTprovides a mechanism for a Point of Analysis (POA) apparatus to accessthe associated asset at a later time if desired.

The method for defining an event of interest results in identificationof a sequence of images which span the event of interest. In anembodiment the sequence of images is compressed with a video compressorcircuit to create the video asset. In an embodiment, some images can bestored in anticipation of the beginning of a event of interest, keepinga constant record of the last several images. This sequence of images isprovided to the compression circuit before the images associated withthe event of interest, providing a short “preroll” of video of theimages leading up to the event of interest. In an embodiment, thesequence of images provided to the compressor circuit can be continuedafter the end of the event of interest to provide a “postroll” of videoof images after the event of interest.

The PORT comprises a bandwidth controller circuit which regulates thearchiving, purging and transmission of assets and references underdirection of a plurality of policies. Policies are selected based on aplurality of conditions including PORT application, date and time,configured bandwidth utilization, PORT status, and network connectivitystatus. A mechanism is provided to allow the POA to change policies andpolicy selection criteria. The PORT contains unique identificationinformation to allow it to be securely and unquestionably associatedwith certain resources on the POA. The PORT also comprises a means forencrypting and signing assets and references independent of datatransport allowing a POA to securely maintain the uploaded content andto validate with a high degree of confidence the providence of theassets transmitted from the PORT.

The PORT comprises means for automatically determining its networkenvironment and contacting the POA with minimal or no userconfiguration. The PORT utilizes only data connection initiated by thePORT to a known location for the POA to function in any local networkwithout user configuration of the PORT or the local network environment.One means is a processor controlled by software to perform networkexploration and self-configuration as disclosed below.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a processor adapted to perform as a circuit according to thepresent invention.

FIG. 2-8 are block diagrams of a point of recordation terminalembodiments.

DETAILED DESCRIPTION OF EMBODIMENTS

The embodiments discussed herein are illustrative of one example of thepresent invention. As these embodiments of the present invention aredescribed with reference to illustrations, various modifications oradaptations of the methods and/or specific structures described maybecome apparent to those skilled in the art. All such modifications,adaptations, or variations that rely upon the teachings of the presentinvention, and through which these teachings have advanced the art, areconsidered to be within the scope of the present invention. Hence, thesedescriptions and drawings should not be considered in a limiting sense,as it is understood that the present invention is in no way limited toonly the embodiments illustrated.

In embodiments of the invention there are circuits for encrypting videoframes at the point of recordation terminal (PORT) apparatus. In anembodiment references, assets, or both are encrypted before storage. Inan embodiment references, assets, or both are encrypted beforetransmission through a public network.

FIG. 1 shows a block diagram of a typical computing apparatus 100 wherethe preferred embodiment of this invention can be practiced. Thecomputer apparatus 100 includes a computer platform having a hardwareunit 103, that implements the methods disclosed below. The hardware unit103 typically includes one or more central processing units (CPUs) 104,a memory 105 that may include a random access memory (RAM), and aninput/output (I/O) interface 106. Various peripheral components such asa camera may be connected to the computer platform 102. Typicallyprovided peripheral components include a data storage device (e.g.flash, or disk) 110 where the policies and images used by the preferredembodiment is stored. A link 112 provides access to the global Internet.An operating system (OS) 114 coordinates the operation of the variouscomponents of the computer system 100, and is also responsible formanaging various objects and files, and for recording certaininformation regarding same. Lying above the OS 114 is a software layer114A. The user layer 114A runs above the operating system and enablesthe execution of programs using the methods known to the art and iswhere most processing as described typically occurs.

An example of a suitable CPU is a Xeon™ processor (trademark of theIntel Corporation); examples of an operating systems is Wind River RTOS.Those skilled in the art will realize that one could substitute otherexamples of computing systems, processors, operating systems and toolsfor those mentioned above. As such, the teachings of this invention arenot to be construed to be limited in any way to the specificarchitecture and components depicted in FIG. 1.

Referring further to the drawings, FIG. 2 is a block diagram of a pointof recordation terminal apparatus. A point of recordation terminalapparatus 200 comprises an asset & event capture circuit 210 comprisinga high resolution digital camera, a video encoding & compressioncircuit, and an image encoding & compression circuit; the asset & eventcapture circuit coupled to a reference selection & meta-tagger circuit220 and the reference selection and meta-tagger circuit couples to aformatting circuit 222 and further coupled to an asset and referencearchive store 240. When it has been determined that the asset capturecircuit has detected an event of interest, an asset is initiated andprocessed as directed by the appropriate policy, normally storing intoan archive. As the asset is being generated, a reference is composed byselecting a representative frame of the video to scale and compress,recording the beginning of the event of interest, and accumulatingrelevant metadata about the generated assets. A processor controlled bycomputer-readable instructions to perform steps of the present inventionis one means for circuits disclosed in this disclosure. In anembodiment, a video encoding and compression circuit is a h.264 encodingcircuit. In an embodiment an image encoding and compression circuit is aJPEG encoding circuit.

Referring to FIG. 3, at least one point of recordation terminal 200 iscoupled to a network 501 through its network interface 230. In anembodiment, the network is a private network. In an embodiment, thenetwork is a wireless cellular network. In an embodiment of the presentinvention, an apparatus comprises a point of recordation terminal 200comprising a network interface 230, the network interface coupled to anetwork 500.

Referring to FIG. 4 The present invention is distinguished by executingpolicies stored in policy store 250 in a bandwidth controller circuit232 to support and mutually benefit the methods of operating a wirelesscellular network. Combined with identity information for the specificPORT, as directed by policy, the reference is transmitted through thenetwork interface 230. Policies also control what is determined to be anevent and control the formatting circuit and control the bandwidthassigned.

Referring to FIG. 5, in an embodiment of the invention, references andassets are encrypted. In an embodiment assets and references areencrypted using a key in encryptor circuit 283 before transmissionthrough the network interface 230. In an embodiment, assets andreferences are encrypted within the point of recordation terminal 200prior to archive store. By using encryption circuits, a public network500 can be used to lower the cost of providing these services. By usingstrong encryption keys with the assets and references, the resultingartifacts can be safely stored in environments with potential securityflaws such as cloud services.

The encryption attached to the assets and reference clearlydistinguishes the present invention from conventional systems which usetransport level security. Once files have completed transport in aconventional system they can be read by anyone representing a continuingloss of privacy for as long as they are stored. In contrast theencrypted assets and references are stored in encrypted format and maynever be decrypted at all before expiration. Because encryption securelyassociates the asset with the device and time of creation, there isprovenance for the assets and references. In an embodiment, each unithas a unique private key of a key pair. It is known that a digitalsignature can establish the source of an image is a specific camera.This can be distinguished from conventional transport level securitywhich does not provide provenance back to the specific PORT and time ofthe event of interest, and creates a security vulnerability as assetsare processed and typically stored in a decrypted format.

Referring to FIG. 6, a method of operating the present invention duringoutage comprises the steps of the connection manager 234 determiningfrom signals of the network interface that the network is congested ordefective, selecting a policy defined for handling assets andreferences, the policy to control:

storing references from the reference selection & meta-tagger circuit220 into the archive store 240,

storing assets into the archive store 240 if there is availablecapacity,

discarding stale assets in the archive store and storing new assets intothe archive store,

discarding new assets if it determines that there is no capacity,

discarding new references if it determines that there is no capacity,and

testing for restoration of network connectivity and improved bandwidth.

Referring to FIG. 6, the present invention comprises a method foroperating an embodiment for recovery after network outage or congestion.In an embodiment, references are stored in the archive store 240 duringnetwork outage or congestion as controlled by the bandwidth controller232. In a non-limiting example, network congestion or outage could causelow or no bandwidth while the PORT 200 is itself operating and detectingevents. When the method determines that the outage is ended or thatbandwidth constraints have loosened, a new policy is selected, whichdirects that references which have been queued in archive 240 betransmitted immediately with no bandwidth restrictions.

Referring to FIG. 7, in normal, unattended operation the operation of aspecific PORT apparatus 200 is as follows: as an assets and eventcapture circuit processes a plurality of images frames and determines ifan event of interest is in progress, a policy based bandwidth controllercircuit 232 selects the normal policy which directs it to store assetsin the archive store 240 and transmit references immediately over thenetwork interface 230 to connected network 500 to the designated POA.

Referring to FIG. 8, while processing, the upload manager 260 monitorsthe status of other elements, in an embodiment including archive store240 to determine if it is nearing capacity and network interface 230 todetermine if it is functioning correctly, and selects alternateprocessing policies for process assets according to the determinedstate. Alternate policies for archive store nearing capacity includedeleting old assets, deleting selected old assets types (video, highresolution snapshots), and not storing new assets. Alternate policiesfor network connectivity are outlined earlier.

If connectivity between a point of recordation terminal and the networkis lost or degraded, but the PORT is otherwise functional, it selects adifferent policy to guide storage and of assets and references.

In an alternative operating mode, the bandwidth controller is configuredwith a policy so that the PORT transmits assets and references as theyoccur. By storing a copy of the transmitted data while the transmissionis occurring, the PORT can provide recovery of data in the event atransmission is determined to be unsuccessful while the transmission isoccurring. When the transmission failure is detected, the remainingportions of the assets and references are generated as normal but nottransmitted. When network connectivity is restored, recovery can beaccomplished as above.

In an embodiment, a POA 300 may request a live feed from a specifiedPORT 200. A live feed is differentiated from the asset and referencemodel in that no event of interest is necessarily involved. Instead thePORT artificially forces an event of interest to be created independentof the content of the images. The artificial event of interest has areference image, typically determined by taking the first image in thesequence. Similarly, the other aspects of the event of interest arecreated independent of the data. If an event of interest does occurduring the sequence of images created by the live stream, it is handledas described in multiple events of interest below. The PORT apparatusresponds to the request for live streaming by selecting an appropriatepolicy, which typically directs the bandwidth controller to allowunlimited transfer of live asset information to the network interface230, passing the processed video information (encoded, formatted, andencrypted) directly to the network interface as it is generated, andindicating to the reference generation circuit that an artificial eventis in progress.

In an embodiment a PORT includes a connection management circuit 234 forinteracting with the POA outside of the upload of assets and references.The connection manager establishes an outbound connection to the POA toallow the PORT to function without requiring any inbound connections.Amongst other things, the connection manager is used to download andmodify policies for the bandwidth controller. The connection manageralso allows the POA to request the immediate upload of a specific storedasset. The PORT responds to this request by immediately transmitting therequested asset under a specific policy, typically no bandwidthconstraints.

It is understood that a network may be a private network, a local areanetwork, a public network, or a combination of the above such as theinternet. Further, the network may be a wireless local network, awireless cellular network, or a wired network. The invention isspecifically distinguished in its ability to function with relativelylow bandwidth and unreliable connections, as typically required for widearea networks, either wired or not. Each point of recordation terminal200 is also coupled to the network by a network interface 230. It isfurther understood that a network in the present patent application isdefined to include proxies, pass-throughs, and other elements which donot change interface modality.

A point of recordation terminal further comprises an asset & eventcapture circuit 210, a reference selection and meta-tagger circuit 220,and an archive store 240, the asset & event capture circuit coupled tothe reference selection & meta-tagger circuit, the network interface 230coupled to the reference selection & meta tagger circuit and coupled tothe archive store.

A PORT further comprises a connection management circuit 234 coupled tothe network interface 230. The connection management circuit establishesa connection to allow configuration and management of the PORT. Becausethe connection management circuit and the transmission circuit disclosedbelow both utilize a connection initiated from the PORT to the POA, theyare compatible with typical network configurations such as NATs (networkaddress translators, which fake a public IP address for a local networkdevice with a private IP address) and Firewalls (which typicallyrestrict almost all inbound traffic but little if any outboundtransactions). In an embodiment, the connection is an HTTP requestinitiated by the PORT which is periodically timed out and re-initiated.If the POA has a configuration directive for the specific PORT inquestion, it responds to the HTTP request with the contents of thedirective. Subsequent requests from the PORT provide the status andresults of the configuration directive. Upon reception of theconfiguration directive, the PORT executes the command and re-initiatesthe connection. In an embodiment, the PORT executes directives with anextend duration by creating a separate process or thread to process thecommand, while re-establishing and maintaining the connection to thePOA, and including in the connection the status of the commandscurrently executing. In an embodiment, when a directive finishes, theHTTP connection is immediately terminated and re-established with thefinal status of the directive, providing immediate feedback of directivecompletion.

An embodiment of the invention is a method comprising the processes of:determining if motion has occurred, defined as an event of interest,defining a small single image to represent the event of interest in atime correlated manner, locally imaging data at all times at allcameras, determining if locally analyzed images are not needed, notrecording or transmitting except for minimal statistics information.

An embodiment of the invention is a process for recognizing an event ofinterest and storing an asset and at least one reference to associate asexemplary of the event. The process comprises known methods for motiondetection, known methods for object detection, and known methods forobject recognition and the following steps: triggering on matching anevent of interest pattern within a certain sequence of images, selectingan exemplary image from the sequence, scaling the exemplary image,compressing the exemplary image, recording the start and end times ofthe event of interest, and additional metadata sufficient to efficientlyprocess and uniquely address the associated asset on the PORT. In anembodiment, an exemplary image is selected from the sequence of imagesin motion as the image with the largest pixel difference from areference image in the sequence. In an embodiment of the method, themethod further comprises operating on the event of interest to generatea high resolution image asset, In an embodiment of the method, themethod further comprises operating on the event of interest to create avery compact image representative. By operating on the event of interestis included the non-limiting examples of no scaling and compression,scaling and compressing in a highly lossy manner, and JPEG encoding. Inan embodiment, the method further comprises the step of recordingadditional metadata derived from the event of interest, by computing theamount of motion detected on each frame and an indication of the currentlogical mode of the motion detection circuit, including preroll, motion,and postroll. In an embodiment the method further comprises creatingreference information for an asset to facilitate the processing orretrieval of assets, in an embodiment the asset size in bytes.

It can be appreciated that the operation on an event of interestdescribed is in anticipation of the POA providing primarily a directuser interface to allow humans to rapidly select events of interest forfurther analysis. In anticipated implementations of a PORT alternativereference and asset information will be captured to allow efficientcomputation processing of references to determine if an event ofinterest requires further analysis, and subsequent processing of theassociated assets. Specifically, is known that object recognitionalgorithms can identify the type of object (such as car, person, face)being imaged and its location. Further it is known additional artifactscan be produced from such object recognition processing, such as thespecific features and their spatial relationship. In support of a POAdoing object processing, a reference would contain limitedcategorization information and the reference would contain the detailedobject features. Thus the PORT architecture of references and assetsshould not be constrained to the specific type of references and assetsdisclosed.

In an embodiment of the invention, the PORT further comprises an assetupload manager circuit. The upload manager circuit functions under aselected policy to send assets to the POA without request from the POA.In anticipation of the POA needing a significant percentage of theassets, and in acknowledgement that a PORT must have limited archivecapacity, the upload manager attempts to send assets proactively to thebandwidth controller 232 for transmission. The bandwidth controllerselects a policy appropriate for the background upload of assets(typically a significantly limited bandwidth allocation) and sends theassets at or below the defined rate. The upload manager also tracks thestatus of reference and assets in the archive, and under policy controlcan immediately delete references and assets once they have beentransmitted, delete them when the archives near capacity, or not deleteselected or all assets. In practice, the events of interest occurinfrequently and have limited duration, the background transmission ofassets can be accomplished in a small fraction of the bandwidth requiredfor transmitting the data in real-time. The upload manager policy can beselected based on conditions including the current status of the archivestore, and the time of day and day of the week. Policies for the uploadmanager include sending all assets in order of storage, sending assetsin reverse order of storage, and sending selected types of assets first,followed by different type of assets. In an embodiment, an uploadmanager circuit comprises a processor coupled to a policy store, thepolicy store comprising computer readable media encoded withinstructions to adapt the processor to perform the above disclosed stepsand processes. The policy store is further coupled to the connectionmanager whereby the contents of the policy store can be initiallyconfigured and updated.

The present invention comprises a computer implemented method forarchive transmittal containing the steps of storing assets locally tothe PORT, tracing assets through a common ID in reference data to allowan arbitrary delay between capture and upload of the asset, limited onlyby the storage available in the PORT. As the bandwidth controllerdetermines that bandwidth is underutilized, assets are transmitted usingthe established reference information to allow a POA to associate theassets uploaded in the this background manner with the originatingreference data. A policy guides but does not dictate the operation ofthe bandwidth controller. In an embodiment, a policy assigns bandwidthby time of day and day of week. In an embodiment, the policy assignsbandwidth during network failures by defining the amount of time to waitin response to a network failure before attempting transmission of astored asset. In an embodiment, the bandwidth controller mayautonomously adapt the policy to use more bandwidth if the asset storeis becoming full.

In an embodiment of an apparatus for documenting at least one occurrenceof an event of interest the apparatus comprises a digital camera coupledto a network interface, the camera, and the network interface coupled tothe following:

a means for determining when an event of interest occurs, in anembodiment a circuit comprising a processor controlled by software toexecute the following computer-implemented steps: motion threshold, and

a means for selecting an extent of data associated with the event ofinterest to accurately represent the event, in an embodiment a circuitcomprising a processor controlled by software to execute the followingcomputer-implemented steps: marking motion activity plus a preroll andpostroll, motion object tracking with analysis artifacts;

a means for efficiently recording the selected extent of data in anembodiment a circuit comprising a processor controlled by software toexecute at least one of the following computer-implemented steps: toh264 encode, to JPEG encode;

a means for storing the recorded events in an embodiment a circuitcomprising a processor controlled by software to execute the followingcomputer-implemented steps: to write to flash memory, to write toSD-card, SDXC-card, SDHC-card or equivalent non-volatile memory card, towrite to disk; and

a means for deriving more compact representations of the event which canassist in determining if the event is of further interest, in anembodiment a circuit comprising a processor controlled by software toexecute the following computer-implemented steps: to generate highlycompressed images, timestamps, motion metadata, and descriptiveinformation for each asset.

In an embodiment, a means for determining when an event of interestoccurs in an embodiment a circuit comprises a processor controlled bysoftware to execute the following computer-implemented steps: todetermine if multiple events of interest occur in close proximity, tocause a single extent of data to be recorded indicative of multipleevent representations, each of which provide indication of where in theextent the event occurred. In an embodiment, if during an event ofinterest or during the postroll period after an event of interest, a newevent of interest is determined to occur, a new set of reference dataand image asset data is generated, and the sequence of images capturedis continued to included the subsequent event of interest. The referencedata and assets have an offset associated with them to indicate at whichnumber in the sequence of images represented by the compressed videothey occur.

In an embodiment, a means for storing recorded events and a means forderiving compact representations comprises a circuit comprising aprocessor controlled by software to execute the followingcomputer-implemented steps: to encrypt data for later decryption

In an embodiment a means for storing the recorded events comprises in anembodiment a circuit comprising a processor controlled by software toexecute the following computer-implemented steps: to store them locallyon the device and to transmit according to a policy implemented in abandwidth controller circuit.

In an embodiment an apparatus for transmitting compact representationsof an event of interest over an unreliable network comprises,

a means for connection comprising at least one of a private network, anIP network, a cellular network, or an IP network over cellular network;

the means for connection coupled to a first network interface circuitand to a second network interface circuit, the first network interfacecircuit coupled to a means for transmission of compact representations,

wherein the means for transmission of compact representations comprisesin an embodiment a circuit comprising a processor controlled by softwareto execute the following computer-implemented steps: to determine if therepresentation cannot be immediately transmitted, to store therepresentations locally and to retry transmission at a later time; and ameans for reception coupled to the second network interface,

A PORT comprises a transmission circuit which transmits data to a POA.In an embodiment, the PORT transmission circuit is a processor adaptedby a software implementation of the HTTP protocol, which initiates aseparate transaction for uploading each set of references and assetsassociated with a event of interest. In an embodiment for streaming, thePORT transmission circuit is adapted to perform the method of the HTTPchunked data transmission model which incrementally transfers largemedia assets as they are generated. In an embodiment, the PORTtransmission circuit maintains a record of data transmitted but notacknowledged by the protocol, and in the event the HTTP transactionfails to complete correctly, the information can be stored in the localarchive for later recovery.

In an embodiment, a PORT further comprises a formatting circuit whichprocesses the compressed video to a format that allows streaming withoutreformatting as well as storage (RTP based protocols allow streaming,MPEG 4 allow storage but not both). In an embodiment the video isformatted in the flash FLV format for H.264 video. In an embodiment, aformatting circuit of the PORT couples to the archive store and to abandwidth controller and to a policy store to interpret a standard H.264bit stream or reference format and convert the data stream directly intothe FLV format while adding minimal (less than a frame) of latency.

In an embodiment, a PORT further comprises a video encoder circuit whichruns constantly, to generate a valid H.264 video stream. In anembodiment, a PORT further comprises a formatting circuit coupled to avideo encoder circuit to detect reference or key frames (I Frames inH.264 nomenclature) and always starts video sequences at I Frameboundaries. In an embodiment, a PORT further comprises a transmissioncircuit which stores a sequence of compressed video frames starting withan I Frame as a preroll buffer, enabling preroll buffering in thecompressed space, significantly reducing the storage required forpreroll.

In an embodiment a point of recordation terminal comprises a circuitcomprising a processor controlled by software to execute at least one ofthe following computer-implemented steps:

to change configuration of other circuits in the terminal,

to transmit immediately when directed by means for analysis,

to store events if immediate transmission fails, and

to specify all data should be recorded and transmitted immediately for alimited period.

In an embodiment the invention comprises a method for operating anapparatus to reliably represent high complexity continuous data over alow bandwidth and unreliable connection, the apparatus comprising:

a point of recordation terminal (PORT) coupled to a connection,

the connection comprising at least one of an IP network, a cellularnetwork, and an IP over cellular network, the method comprisescapturing, and transmitting an event of interest, wherein capturing anevent of interest comprises the following processes:

determining when an event of interest occurs,

selecting an extent of data associated with the event of interest,

efficiently recording the selected extent of data,

deriving a compact representation of the event of interest, and

storing the recorded events; wherein transmitting an event of interestcomprises the following processes:

transmitting immediately when directed and storing if immediatetransmission fails,

opening an client session to a server at a designated address

transmitting data over the session,

maintaining a record of data transmitted but not acknowledged,

recording the record of data in the event the transmission session fails

storing recorded events locally and transmitting when an acceptableamount of bandwidth becomes available, and

responding to subsequent request to immediately transmit a stored recordby transmitting the data rapidly.

In an embodiment a point of recordation terminal apparatus comprises:

a high resolution digital camera,

a first storage device,

a first network interface,

a circuit controlled by software to signal a motion threshold,

a circuit controlled by software to measure motion activity,

a circuit controlled by software to track motion objects with analysisartifacts,

a circuit controlled by software to encode h.264 format files,

a circuit controlled by software to encode JPEG files,

a circuit to write to flash memories (as non-limiting example an SDcard),

a circuit to generate small reference images, timestamps, motion andasset meta-data,

a circuit controlled by software to determine if multiple events ofinterest occur in close proximity,

a circuit controlled by software to cause a single extent of data to berecorded indicative of multiple event representations and where theyoccur,

a circuit controlled by software to store assets locally on the firststorage device and to retry transmission at a later time wherein

a circuit comprises a processor controlled by software instructions andthe processor is coupled to the first network interface, the processoris coupled to the first storage device, and the processor is coupled tothe high resolution digital camera.

In an embodiment of the invention, a PORT further comprises a policystore, the policy store coupled to the connection manager, a formattingcircuit, the formatting circuit coupled to the policy store, to thearchive store, to the upload manager, and to the bandwidth controllerand to the video encoding circuit. In an embodiment of the invention, aPORT further comprises an upload manager circuit coupled to the archivestore and to the policy store, and to the bandwidth controller. In anembodiment of the invention, a PORT further comprises a connectionmanager coupled to a policy store and coupled to the network interface.

In embodiments of the invention, a PORT further comprises at least oneof a policy store, a connection manager, a formatting circuit, and anupload manager. The policy store is coupled to the connection manager,the formatting circuit, the upload manager, the bandwidth controller,and the asset and event capture circuit. The formatting circuit isfurther coupled to the video encoding circuit, to the archive store, tothe bandwidth controller. The upload manager circuit is further coupledto the archive store, and to the bandwidth controller. The connectionmanager is further coupled to the bandwidth controller and to thenetwork interface. Therefore, policies which determine actions uponcertain conditions are received from the network by the connectionmanager and stored to the policy store whereby the upload managercircuit determines which and how quickly assets are transmitted via thebandwidth controller and the network interface, whereby the formattingcircuit determines how to convert raw video to streamable video and howto determine the preroll and post roll parameters, whereby theconnection manager changes the operating mode upon certain conditionsspecified in a policy stored in the policy store.

In an embodiment, a PORT provides metadata captured outside of events ofinterest which represents the basic inputs to the event of interestdetermining circuit. In an embodiment a PORT periodically uploads thisinformation as it is generated. The upload of this information allows aPOA to analyze the PORT configuration to determine if some otherconfiguration would better capture appropriate events of interest. Theperiodic upload allows the POA to determine the basic operational statusof a connected PORT.

A bandwidth controller circuit executes a first bandwidth managementpolicy for the upload of references and a second bandwidth managementpolicy for the upload of assets. Different modes distinguish “real time”and the recovery mode. The bandwidth controller circuit implementsretention policies for both on camera assets, and on camera references.In an embodiment, if a camera runs out of space, the bandwidthcontroller circuit determines what to throw away (in an embodiment itthrows away complete asset sets for oldest events), but it can do otherthings—throwing away “snapshots” but keeping the video for example.

In an embodiment a bandwidth controller is set to one of severalpolicies in the event of losing network connectivity, such as thenon-limiting exemplary policies: storing for recovery and justdiscarding. A service provider offers additional capacity at incrementalpricing. In an embodiment the PORT self regulates its uploading of anasset according to its embedded policy. In an embodiment a serverremoves bandwidth limitation for a specific asset (and no othertransfer) and demands that asset be uploaded without delay. Accordingly,the PORT records such a demand upload and removes it from the queue ofassets remaining

In an embodiment, a PORT receives a policy conditioned on whether acamera has storage available and on whether services have been selectedfor subscription. Specifically an SD card slot in the camera enablesbandwidth shaping. In an embodiment data on the SD card is independentlyavailable without decryption. In an embodiment data on the card isstored encrypted.

In an embodiment, the bandwidth controller is a processor controlled bysoftware for policy management for to determine when to upload and howmuch. In an embodiment it utilizes time-of-day (e.g. don't contend forinterne connection when customers are using wi-fi service, but changebandwidth limits after midnight). In an embodiment it utilizesreliability measurements (if packet loss on the link exceeds athreshold, back off sending for a random or fixed time amount to reducecontention). In an embodiment the bandwidth controller circuit utilizespricing models to determine when to upload and how much (e.g. ifunlimited connectivity on my wireless plan after 7, only send referencesthen). In an embodiment, a set of PORTs are organized as a group andbandwidth policy is managed among the group.

An apparatus for generating and storing an asset comprises a digitalcamera coupled to video memory, the memory coupled to an archive storesuch as a removable SD flash memory card, and a processor coupled to allthe above and to a network interface card.

One means for reading and encoding a camera identification is aprocessor encoded with a PrettyGoodPrivacy strong encryption algorithmand a private key. One means for reading and encoding a time of day ofthe asset is reading Unix time from an internet server at the time thefirst video frame is captured by a digital camera attached to aprocessor. One means for selecting and storing at least one highresolution digital photograph is a motion detection circuit coupled to amemory configured as a pipeline coupled to a digital camera. Anothermeans is comparing each digital camera frame to a reference frame andcapturing a frame having a number of pixels above a threshold differentfrom the reference frame. One means for deriving and storing a mediumresolution video image sequence is a jpeg or mpeg chip coupled to avideo memory and writing to a flash memory. One means for reading andencoding at least one offset of at least one high resolution digitalphotograph relative to the time of day of the asset is subtracting thetime of the start of the asset from the time at the threshold crossingframe.

The apparatus comprises a circuit coupled to a video memory and writingan asset to a flash memory wherein the asset is an encrypted digitalfile.

One means for determining and encoding a type of event is reading fromthe threshold circuit comparing a reference frame to a video frame theparameters of difference. One means for computing and storing a digitalsignature is encoding a processor with a PrettyGoodPrivacy algorithm andcombining a private key, the time of day of the asset, and the size ofthe asset or reference. One means for determining and storing a prerollbefore the start of the event is counting the stages of a pipelinememory from the entrance until the point that an event has beendetermined. One means for determining and storing a postroll after theend of the event is adding a fixed value to the time of the end of theevent.

The apparatus comprises a processor adapted to read a video memory andgenerate a reference which is an encrypted digital file.

One means for deriving a low resolution, scaled still image is encodinga processor with a JPEG algorithm, reducing the scale of a photograph toless than 100.times.100 pixels, and setting the JPEG algorithm to lowresolution. One means for reading and storing a size of the asset isinstructing a processor to read the file header from the flash memorycontroller.

One means for deriving meta-data values includes a processor readingoutput values from a circuit for graphics processing coupled to a videomemory.

Said means comprises a circuit comprising a processor coupled tocomputer-readable media encoded with instructions for computingmeta-data values, determining the size of an asset, determining anevent, selecting a high resolution digital photograph from an imagesequence, converting an image sequence into a medium resolution videoimage sequence, deriving a compressed, scaled, low resolutionrepresentation from a selected high resolution digital photograph,reading camera identification and computing a digital signature, whereina reference comprises a plurality of digital files encoded by strongencryption.

Means for reading and encoding a PORT identification include a processorencoded to perform a digital signature on an encoded image using aprivate key unique to the PORT.

Means for generating a PORT unique identification for the asset includea processor encoded to

increment an event number, or

encode the time and date of the event.

Means for generating multiple representations of the event include aprocessor encoded to:

include an encoded video representation of an image sequencerepresentative of the event,

wherein an image sequence representative of an event includes imagesfrom immediately before the event of interest,

wherein an image sequence includes image from immediately after theevent of interest;

to indicate the relative activity detected in each image of thesequence;

to include data derived from analysis of the event of interest;

to include an encoded high resolution image of an image representativeof the event;

to reference two grouping of representations, one optimized forminimizing the number of bytes required and one optimized to accuratelyrepresent the event of interest;

to identify two groups associated by the unique identifier,

wherein one of the two groups provides indication of the exactrepresentations in the available in accurate representation, wherein oneof the two groups includes size, relationship, and type indication; and

to combine representations into a single larger group if two events ofinterest occur sufficiently close in time that events immediately beforeor after would overlap.

Means for indicating the timing relationship between differentrepresentations include a processor encoded to:

record the sequence number of the image from the start of therepresentation, or

record the time and data of the representation.

An apparatus is disclosed comprising a digital camera coupled to aformatting circuit coupled to an encryption circuit coupled to anarchive store, wherein the encryption circuit comprises an input forreading a unique camera identification key, an input for reading a videostream from the formatting circuit, a processor for encoding the videostream with time, date, and the unique camera identification key, and anoutput for writing the resultant encoded video stream to the archivestore.

An apparatus is disclosed comprising a digital camera coupled to areference select & meta-tagger circuit coupled to a formatting circuitcoupled to a connection manager circuit coupled to a network interface,wherein the connection manager circuit comprises a processor controlledby software to perform the following operations: reading a destinationIP address hardcoded onto the connection manager circuit board,receiving a compact representation of an event of interest from thereference select & meta-tagger circuit, preparing packets with thedestination IP address containing the compact representation, opening aclient session with the destination IP address, and transmitting thepacket as a client to a server at the destination IP address.

A point of recordation terminal apparatus is disclosed comprising:

a network interface, the network interface coupled to a network;

an asset & event capture circuit;

a reference selection & meta-tagger circuit;

a bandwidth controller circuit; and

an encryption circuit, whereby captured assets and references areencrypted prior to transmission.

The apparatus further comprises an archive store coupled to theencryption circuit whereby captured assets and references are stored inencrypted form into the archive store.

The encryption circuit is uniquely associated with the specific PORT bycryptographic operation. The encryption circuit indicates the time anddate of the event of interest by cryptographic operation on the assets.

A method is disclosed comprising transmitting a reference immediatelywhile storing an asset into the archive store. The method furthercomprises temporarily storing the transmitted reference and storing itto the archive store in case the transmission fails.

By storing is meant the steps of detecting when the transmission islikely to be possible again and retransmitting the reference.

A point of recordation terminal apparatus is disclosed comprising:

a network interface, the network interface coupled to a network;

an asset & event capture circuit;

a reference selection & meta-tagger circuit;

wherein the network interface comprises a configuration detectioncircuit whereby it automatically detects and configures its networkinterface settings.

A method of operating the configuration detection circuit is disclosedcomprising sequentially trying the following processes until a workingconfiguration is established: DHCP, static configuration andauto-detection, wherein auto-detection comprises

determining the local addressing scheme;

selecting a host address not detected in the local network, probing theselected address to determine if used, and reselecting if collision isdetected; and

sending a prospective transaction on at least one port to eachidentified hosts on the local network to determine if any act as agateway, and selecting a host as a gateway if successful.

In an embodiment, determining the local addressing scheme comprisespassively listening to network traffic to determine the local addressingscheme and hosts on the networks. In an embodiment, determining thelocal addressing scheme comprises actively probing the network todetermine the local addressing and hosts on the local network.

A point of recordation terminal apparatus is disclosed comprising

a network interface, the network interface coupled to a network;

an asset & event capture circuit;

a reference selection & meta-tagger circuit; and

a connection manager circuit, whereby the connection manager and thenetwork interface establish client sessions to a server at a knownlocation.

Methods of operating the apparatus include without limitation thefollowing independent processes:

establishing an HTTP or HTTPS protocol client session.

receiving commands issued by a server responding to a client.

periodically reestablishing its client connection to a server.

processing a command to quickly reestablish a client connection.

providing status indication for commands currently running in a clientconnection and for commands recently completed in a client connection;and other methods for operating the apparatus known in the art.

CONCLUSION

The present invention is distinguished from conventional videosurveillance systems by using a public network enabled by its bandwidthcontroller and encryption circuits, by providing for low bandwidthreference transmission in near real time while queuing multi-frameassets for policy controlled transmission, and policy controlledbandwidth control in response to recovery, normal operation, streaming,and searching.

The present invention is distinguished from conventional cameras bydetermining if motion has occurred within a period, creating at leastone reference indicative of the motion, transmitting the references inreal time, and only storing, analyzing, or uploading data around timesof motion to reduce bandwidth consumption. In particular, the inventionallows efficient and secure use of cloud computing. By encrypting assetsand references on a per PORT and per user basis and not decrypting themduring upload and storage, the security and providence of the data isassured even when using resources shared across many differentcompanies. The PORT is distinguished from convention video cameras byusing only outbound network connections compatible with a wide areanetwork to establish connection with a POA. It is particularly pointedout and distinctly claimed that a network can connect using a cellularnetwork as the back haul as the disclosed bandwidth utilization modelmakes it practical and affordable (since cellular bandwidth is veryexpensive compared to landline/wi-fi).

Significantly, this invention can be embodied in other specific formswithout departing from the spirit or essential attributes thereof, andaccordingly, reference should be had to the following claims, ratherthan to the foregoing specification, as indicating the scope of theinvention.

1. A point of recordation terminal (PORT) apparatus comprising: anetwork interface, the network interface coupled to a network, an asset& event capture circuit, a reference selection and meta-tagger circuit,and an archive store, the asset & event capture circuit coupled to thereference selection and meta-tagger circuit, the network interfacecoupled to the reference selection & meta tagger circuit and coupled tothe archive store.
 2. The apparatus of claim 1 wherein a network is aprivate network.
 3. The apparatus of claim 1 wherein a network is acellular network.
 4. The apparatus of claim 1 wherein a network is apublic network and the point of recordation terminal further comprisesan encrypt circuit wherein the encrypt circuit couples the referenceselection and meta-tagger circuit and the archive store to the networkinterface of the point of recordation terminal, whereby references aretransported through the public network and stored in encrypted form andonly decrypted upon request of a user.
 5. A method for operating a PORTapparatus comprising: analyzing video imagery for motion matchingcertain behaviors, recording imagery around an event consisting ofminimal reference information and larger richer assets, storing richerassets locally if it is determined that it is inefficient to attempttransmission immediately and trickling assets when bandwidth is lessconstrained.
 6. A system for providing security as a service to aplurality of locations by network distributed video surveillanceequipment, comprising an apparatus and computer-implemented method foroperating the apparatus, wherein said apparatus comprises a point ofrecordation terminal (PORT) apparatus, the PORT apparatus coupled to alocal area network and having an archive store; and at least one circuitfor encrypting data; wherein said computer-implemented method foroperating the apparatus comprises: encrypting a reference and an assetwith a private key prior to transmission via the local area and publicnetworks; wherein an asset comprises a plurality of video frames and areference comprises a selected video frame selected from an asset, time,date, and identity of the PORT apparatus on which it was recorded.
 7. Amethod of operating an apparatus for providing video surveillance as anetwork hosted service to a plurality of small and medium enterprises,the apparatus comprising at least one point of recordation terminal(PORT) apparatus, each PORT apparatus coupled to a network, the methodcomprising: capturing a plurality of video frames, storing a pluralityof video frames in an archive store; determining an event occurrence;selecting at least one video frame, time, date, and identification as atleast one reference to a video sequence whereby each reference isindicative of where the video frame occurs in the video sequence; andtransferring the reference through a network.
 8. A point of recordationterminal (PORT) apparatus for to capture, select, and transfer videoframes and related data on a network without clogging local networktraffic, the PORT apparatus comprising a processor and memory; a digitalcamera; a video encoding & compression circuit; an image encoding &compression circuit; an event determination circuit; a video buffer; areference selection circuit; an archive store; a private key encryptioncircuit; a bandwidth controller circuit and a network interface.
 9. Apoint of recordation terminal comprising a processor controlled bysoftware to execute at least one of the following computer-implementedsteps: to change configuration of other circuits in the terminal, totransmit immediately when directed by means for analysis, to storeevents if immediate transmission fails, and to specify all data shouldbe recorded and transmitted immediately for a limited period
 10. A pointof recordation terminal apparatus comprising: a network interface, thenetwork interface coupled to a network; an asset & event capturecircuit; a reference selection & meta-tagger circuit; wherein thenetwork interface comprises a configuration detection circuit whichautomatically detects and configures its network interface settings. 11.A point of recordation terminal apparatus comprising a networkinterface, the network interface coupled to a network; an asset & eventcapture circuit; a reference selection & meta-tagger circuit; and aconnection manager circuit, whereby the connection manager and thenetwork interface establish client sessions to a server at a knownlocation.
 12. A method for operation of a configuration detectioncircuit of a point of recordation terminal apparatus comprising:sequentially trying the following processes until a workingconfiguration is established: DHCP, static configuration andauto-detection.
 13. The method of claim 12 wherein auto-detectioncomprises determining the local addressing scheme; selecting a hostaddress not detected in the local network, probing the selected addressto determine if used, and reselecting if collision is detected; sendinga prospective transaction on at least one port to each identified hostson the local network to determine if any act as a gateway, and selectinga host as a gateway if successful.
 14. The method of claim 13 whereindetermining the local addressing scheme comprises passively listening tonetwork traffic to determine the local addressing scheme and hosts onthe networks.
 15. The method of claim 13 wherein determining the localaddressing scheme comprises actively probing the network to determinethe local addressing and hosts on the local network.